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ABSTRACT 

This  interim  technical  report  describes  the  work  performed  in  Phase  III 
of  Contract  P33615-82-C-5114  covering  the  period  1  May  1983  to  30  September 
1983.  As  a  result  of  this  investigation,  we  recommend  that  an  intelligent 
manual  expert  system  be  implemented  to  assist  in  the  price  analysis  phase 
of  the  procurement  process.  This  recommendation  is  based  upon: 

1.  A  review  of  training  programs  and  authoritative  publications. 

2.  Briefings,  interviews  and  case  analyses  performed  by  personnel 
from  three  Air  Force  procurement  organizations. 

3 .  An  evaluation  of  three  alternative  system  designs. 


A  prototype  intelligent  manual  was  constructed  using  ZOG,  an  information 
organization  system.  The  system  architecture  was  determined  by  a  task 
analysis  of  the  pricing  task.  The  pricing  task  was  identified  by  field 
investigations  and  was  found  to  be  a  relatively  small  component  of  the 
procurement  task.  This  coupled  with  the  distributed  nature  of  the 
requisite  information  render  implementation  of  a  more  complex,  self- 
contained  expert  system  infeasible  at  the  present  time.  Even  though  the 
intelligent  manual  lacks  decision-making  capabilities,  its  implementation 
would  allow  data  to  be  gathered  and  technology  to  be  developed  and 
disseminated  that  would  be  necessary  for  constructing  more  self-contained 
systems. 


The  implications  for  future  research  such  as  human  interface  issues  and 
system  application  and  extention  are  discussed.  The  limitations  of  the 
recommended  approach  such  as  the  inability  of  the  system  to  draw  inferences 
from  prior  decisions  and  the  restricted  focus  of  the  study  are  identified 
and  several  alternative  designs  are  evaluated. 
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1.  INTRODUCTION 


1*1  Sosaary  of  Prior  Report 

The  previous  technical  report  described  the  work  accomplished  during 
Phases  I  and  II  of  this  project  (covering  the  period  from  1  October  1982  to 
30  April  1983).  It  contained  conclusions  about  the  feasibility  of  expert 
systems  for  price  analysis,  provided  an  analysis  of  contract  pricing, 
discussed  the  preliminary  design  of  an  intelligent  manual,  and  briefly 
surveyed  the  educational  environment  within  the  Air  Force.  The  report 
identified  problems  associated  with  price  analysis  during  procurement  and 
discussed  the  feasibility  of  designing  an  "expert  system"  using  techniques 
from  the  field  of  artificial  intelligence.  The  analysis  suggested  that  an 
"intelligent  manual"  type  of  expert  system  would  be  most  useful.  The 
maximum  benefit  of  this  system  would  be  felt  in  base- level  procurement. 

The  design  of  a  prototype  of  the  intelligent  manual  in  an  information 
base  called  XIHFO*  was  presented.  The  problems  with  this  design  were 
discussed  and  an  initial  redesign  using  another  information  organizing 
2 

system  called  ZOG  was  presented. 

1.2  Preview  of  this  report 

This  technical  report  describes  the  work  accomplished  during  Phase  III 
of  the  project.  In  addition,  we  report  on  activities  performed  as  part  of 
Phases  I  and  II  not  previously  reported. 

These  activities  include: 


^XINFO  is  not  an  abbreviation;  the  name  is  intended  mainly  a  mnemonic  for 
the  extended  information  base  capabilities  of  the  system. 


^ZOG  is  not  an  abbreviation,  either.  The  name  ZOG  was  chosen  to 
represent  the  novelty  of  the  ZOG  style  of  human-computer  interaction. 
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1.  Interviews  and  briefings  with  procurement  personnel  designed  to 
identify  existing  Air  Force  data  bases  (as  required  by  the 
Statement  of  Work.  (SOW)  4.1 .2.4). 

2.  Defining  and  documenting  rules  and  experience  used  in  the  Air 
Force  (SOW  4.2.2). 

3.  Developing  a  model  of  the  methodological  approach  used  by  the 
Air  Force  procurement  personnel  based  upon  the  findings  (SOW 
4.2.1). 


The  tasks  performed  during  Phase  III  reported  herein  include: 


1.  Developing  the  expert  computer  system  using  the  knowledge  base 
developed  during  the  data  gathering  phase  (SOW  4.3.1). 

2.  Building  a  demonstration  prototype  of  the  system  within  the  ZOG 
framework  (SOW  4.3.2). 

3.  Defining  areas  of  future  research  and  identifying  limitations  of 
the  proposed  system  (SOW  4.3.3). 

4.  Discussing  alternative  system  designs  (SOW  4.3.4.). 
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2.  CONSTRUCTION  OF  KNOWLEDGE  BASE  (SOW  4.1 .2.4) 

In  cbis  chapter  ve  identify  existing  data  bases  in  the  United  States  Air 
Force  (USAF)  that  may  be  used  in  analyzing  contractor  cost  and  price 
proposals.  We  define  the  scope  of  these  data  bases  and  their  uses  in  the 
contract  pricing  environment. 


2.1  Current  Data  Bases 

The  following  online  data  bases  related  to  the  procurement  process  were 
identified: 


1.  CLAPS  —  used  at  the  base  level.  This  database  contains  a 
classified  index  of  items  purchased  in  the  past  and  contains 
information  about  prior  buys  as  well  as  local  dealers  and 
manufacturers  of  the  item. 

2.  J041  —  used  by  the  Air  Logistics  Cossnand.  This  database  is 
largely  a  management  tool  used  to  track  the  activity  of  the 
procurement  section  with  respect  to  individual  purchases. 

3.  Government  Services  Administration  schedules  —  These  schedules 
document  agreements  between  a  large  number  of  contractors  and 
the  US  Federal  government  as  a  whole.  Since  these  schedules 
specify  prices,  the  buyer  can  use  them  as  a  substitute  for 
obtaining  an  actual  quote. 

4.  Price  indices  —  These  are  usually  issued  monthly  and  annually 
by  the  Department  of  Commerce  and  are  specific  to  different 
areas  of  the  country.  They  can  be  used  to  adjust  historical 
pricing  data. 

5.  On-site  databases  —  provided  to  the  Air  Force  Systems  Command 
and  the  Defense  Logistics  Agency  (DLA).  AFPRO  and  DLA 
representatives  at  the  contractor's  manufacturing  site  have 
access  to  historical  and  current  information  about  the 
contractor's  actual  costs  and  practices.  This  is  useful  for 
evaluating  sole-source  proposals. 

6.  AMIS  —  used  by  Air  Force  Systems  Comamnd.  This  system 
(Acquisition  Management  Information  System)  provides  detailed, 
historical  procurement  information  on  over  61,000  contracts  as 
well  support  for  contract  preparation  and  report  generation. 

These  are  historical  data  bases  that  contain  some  procurement  history  on 
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prior  buys.  To  the  extent  that  historical  information  is  useful  in 
establishing  a  government  position,  these  data  bases  are  useful.  However, 
their  usefulness  is  degraded  by  the  time  lapsed  between  the  last  purchase 
and  the  current  one,  by  the  inability  to  identify  the  same  or  similar  items 
within  the  system,  by  the  time  it  takes  to  get  the  prior  procurement  into 
the  system,  and  by  the  low  commitment  of  personnel  to  maintaining  these 
systems.  Prior  contract  files  maybe  available  in  hard  copy  but  they  are 
neither  easily  accessible  nor  consistently  documented. 

The  Contract  Data  and  Management  Systems  (CDMS)  section  in  the  Air  Force 
Logistics  Command  appears  to  be  in  the  process  of  implementing  a 
coordinated  data  base  management  program  that  will  incorporate  and  replace 
the  current  data  management  programs  for  Air  Force  Logistics  Centers 
(ALCs).  This  system  would  provide  the  historical  procurement  data  on-line 
to  all  ALCs. 

The  base  level  buyers  use  the  historical  information  as  the  primary 
basis  for  their  objective  price  if  it  is  available  and  if  they  have  no 
diaconf irming  evidence.  Air  Force  Logistics  Command  (AFLC)  buyers  use 
historical  cost  as  a  last  resort  for  establishing  an  objective  price 
because  of  the  dynamic  nature  the  procurement  environment.  Air  Force 
Systems  Command  (AFSC)  analysts  are  generally  leery  of  using  historical 
costs  because  of  the  technological  environment  and  the  uniqueness  of  most 
of  their  procurements. 

2.2  Interviews  and  Briefings  (SOW  4.2.1) 

In  this  section  we  describe  the  procedures  and  techniques  followed  in 
defining  the  Air  Force  experts'  knowledge  base. 

The  following  interviews  were  undertaken  in  addition  to  those  reported 
in  the  first  Technical  Report. 

1.  The  research  group  met  with  Mr.  Rollie  McReynclds  of  AFLC  at 

AFLC  Headquarters,  Wright-Pat  ter son  Air  Force  Base  (VPAFB).  The 
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in-depth  interview  covered  issues  with  respect  to  automation  of 
the  AFLC  buying  activities  and  Mr.  McReynold's  reaction  to  a  set 
of  base- level  pricing  cases.  The  topics  discussed  included  the 
Pacer  Price  program,  provisioning  and  replenishment  spares 
procurement,  and  sampling  techniques  used  in  pricing.  The 
primary  focus  of  the  meeting  was  to  have  Mr.  McReynolds  evaluate 
a  set  of  base  level  pricing  cases  from  two  aspects.  First,  Mr. 
McReynolds  analyzed  the  cases  to  determine  the  adequacy  of  the 
case  packet.  Mr  McReynolds  evaluated  six  case  packets.  The 
indepth  interview  was  very  helpful  in  identifying  the  process 
that  a  buyer  must  undertake  in  evaluating  a  case  and  determining 
the  extent  of  incompleteness  of  the  case  packets,  and  the 
information  necessary  to  complete  them. 

2.  The  research  group  met  with  two  base  level  supervisors  from  the 
2750th  Air  Base  Wing  (ABW)(Ms.  Viola  Williams  and  Mr.  Marty 
Spaare)  and  with  two  Air  Base  Logistics  price  analysts  (Mr.  Bill 
Bower  and  Mr.  Charles  Warren)  who  were  responsible  for  the 
training  of  the  2750th  base  level  buyers.  Each  supervisor  and 
analyst  met  separately  with  one  member  of  the  research  group. 
During  these  sessions,  they  were  given  case  packets,  one  at  a 
time,  and  told  to  "think  aloud"  while  they  came  to  their 
determination  of  a  fair  and  reasonable  price  for  the  item  being 
purchased.  The  sessions  were  recorded  on  tape  and  were 
subsequently  transcribed.  These  sessions  provided  excellent 
insight  into  the  base  level  buying  process.  The  interviews  were 
conducted  at  WPAFB. 

3.  The  research  group  received  a  briefing  from  Mr.  Roy  Bondurant, 
CDMS,  AFLC,  WPAFB  to  acquaint  them  with  the  current  data 
management  capabilities  of  AFLC  and  the  plans  for  the  future. 

The  current  capabilities  are  not  adequate  and  HQ  has  initiated 
programs  to  correct  the  deficiencies.  The  major  thrust  of  these 
programs  is  to  consolidate  data  management  into  one  integrated 
system.  The  goal  of  the  program  is  to  have  the  ALCa  automated 
by  the  mid-1980s  with  extensive  online  capabilities.  This 
includes  providing  buyers  with  easy  access  to  terminals  and 
suggests  that  a  work  station  concept  is  a  viable  alternative 
that  is  currently  being  considered  by  the  AFLC.  If  this  is  the 
case,  it  would  be  feasible  to  implement  an  initial  version  of  an 
intelligent  manual  as  part  of  such  a  system.  This  is  a  possible 
way  to  test  our  proposed  system  in  a  more  realistic  setting. 

4.  We  met  with  Base  Level  personnel  from  the  2750th  Air  Wing, 

WFAFB.  We  met  first  with  Mr.  Bill  Bower,  a  price  analyst  for 
the  2750th  at  WPAFB.  We  discussed  the  case  packets  and  the 
possibility  of  receiving  additional  data.  He  stated  that  some 
of  the  needed  data  was  currently  available  and  he  would  attempt 
to  find  the  remainder.  He  provided  us  with  a  list  of 
information  supplied  to  the  buyer  by  the  supply  system.  We  next 
met  with  Ms.  Barbara  Walls,  Chief,  System  Management  Section, 
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who  has  had  many  years  of  buying  experience  and  is  currently  in 
charge  of  the  Customer  Integrated  Automated  Purchasing  System 
(CIAPS)  program.  Ms.  Walls  indicated  the  information  was 
supplied  to  the  buyer  and  the  portion  of  that  information  was 
provided  by  an  automated  data  base.  The  CIAPS  system  was 
explained  as  part  of  this  discussion.  She  also  explained  the 
GSA  catalog  system  and  gave  a  demonstration. 

5.  Two  base  level  buyers  from  the  2750th  were  observed  by 
individual  members  of  the  research  team  as  they  carried  out 
their  work  tasks.  Each  was  questioned  as  to  what  tasks  were 
being  performed  and  the  reasons  for  the  actions  being  taken. 

The  buyer  discussed  the  cases  in  which  she  was  currently 
involved  pointing  out  relevant  issues  and  explaining  their 
anticipated  resolution.  They  were  also  questioned  about  the 
reference  material  they  used  and  the  assistance  provided  by  the 
price  analysts.  These  interviews  were  helpful  in  gaining  an 
appreciation  for  the  environment  in  which  a  buyer  must  work. 

6.  Professors  Dillard  and  Ramakrishna  accompanied  by  Major  Weber, 
visited  buying  activities  in  the  San  Antonio  Texas  area.  Visits 
were  made  to  the  ALC  at  Kelly  AFB,  AF  Training  Command,  Randolf 
APB,  and  the  San  Antonio  Contracting  Center.  The  following 
activities  were  accomplished: 

a.  Case  evaluation  sessions  with  two  ALC  buyers (Ms.  Anita 
Maldonado  and  Mr.  Billy  W.  Sullivan)  and  two  ALC  price 
analysts  (Mr  Earl  K.  Booth  and  Mr.  Leroy  Hassler).  These 
personnel  were  presented  with  the  cases  packets  and  asked 
to  "think  aloud"  as  they  analyzed  the  cases  in  an  attempt 
to  determine  the  fairness  and  reasonableness  of  the 
contractors  proposed  price.  The  protocols  used  and  the 
results  obtained  were  similar  to  those  presented  above. 

The  ALC  procurement  activities  appeared  to  be 
characterized  by  more  buyer  specialization  and  better 
support.  This  is  the  result  of  the  higher  dollar  values 
of  the  buys  and  the  higher  volume  of  specific  types  of 
buys  . 

b.  As  was  the  case  with  the  base  level  buyers,  the  two  ALC 
buyers  (Ms.  Particia  Larzelere  and  Ms.  Rita  Brown)  were 
observed  for  the  greater  part  of  a  day  as  they  carried  out 
their  normal  daily  tasks.  This  allowed  us  to  identify 
informal  information  files  and  certain  heuristics  used  by 
the  buyer.  For  example,  one  of  the  buyers  maintained  a 
file  of  "unique  situations"  in  which  she  had  been 
involved.  She  also  maintained  items  which  she  perceived 
to  be  useful  in  training  new  buyers.  The  importance  of 
considering  the  total  procurement  task  was  pointed  up. 
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c.  We  received  an  explanation  and  demonstration  of  the  new 
Automated  Contract  Writing  System  being  installed  at  the 
Kelly  ALC.  The  demonstration  was  conducted  by  Ms. 

Margaret  Comas. 

d.  We  met  with  the  following  contracting  personnel  in  the  AF 
Training  Command  who  were  involved  in  base  level 
procurement:  Ms.  Reinette  Alecozay,  Mr.  John  Elliot, 
Leslie  Kempler,  and  MSgt.Kurt  L.  Stellman.  We  discussed 
pricing  problems  and  their  reaction  to  the  ideas 
concerning  an  expert  system  for  assisting  the  buyer.  The 
price  analysts  present  commented  on  the  major  problems 
faced  by  ATC  base  buyers. 

e.  We  met  with  the  following  personnel  at  San  Antonio  Central 
Contracting  Center  (a  central  base  procurement  facility 
for  the  five  military  bases  in  the  area):  Ms.  Yvonne 
Zakrzevski,  Mr.  Isidoro  Leds,  Jr.,  Ms.  Stehanie  Apple,  Ms. 
Myrna  Howell,  Mr.  Qnil  Kirbery.  We  discussed  the  problems 
they  were  having  and  possible  ways  that  an  expert 
knowledge  based  system  might  be  implemented.  These  buyers 
saw  little  that  could  be  gained  by  such  a  system.  This 
pointed  out  the  critical  importance  of  human  factors 
issues  in  implementing  and  operationalizing  such  a  system. 
We  toured  their  facility  and  briefly  observed  their 
operational  environment. 


2.3  Cases 

Generally,  the  case  analysis  sessions  were  carried  out  by  having  a 
buyer,  price  analyst,  or  supervisory  personnel  "think  aloud"  while 
analyzing  actual  base  level  procurement  cases.  The  cases  were  provided  by 
the  USAF  and  purported  to  be  representative.  The  research  team  was  not 
allowed  direct  access  to  the  case  files.  The  cases  were  current,  with  the 
oldest  one  being  approximately  two  years  old.  The  type  and  number  of  cases 
in  the  packets  and  a  description  of  a  typical  packet  are  presented  in 
Appendix  III. 

2.4  Protocol  for  Interviews 

The  interview  sessions  were  organized  in  two  parts.  In  the  first  part 
of  the  session,  after  explaining  the  purpose  of  the  interview,  the 
?">- urement  person  (buyer,  price  analyst,  or  supervisor)  was  asked  to 
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provide  demographic  information  such  as  the  type  and  length  of  procurement 
experience,  current  position  and  grade,  specialties,  training  and  career 
plans.  They  were  then  asked  some  general  questions  about  the  procurement 
tasks  which  they  performed.  They  were  then  provided  with  the  instructions 
(see  Appendix  17)  for  performing  the  second  phase  of  the  interview  session, 
the  case  analysis  phase.  The  interview  sessions  were  recorded  on  tape  and 
transcribed  for  analysis.  Due  to  the  inadequacy  of  the  case  packets,  the 
formal  experimental  verbal  protocols  were  not  obtainable.  However,  the 
indepth  interviews  that  were  undertaken  were  considered  to  be  adequate  for 
obtaining  the  requisite  information.  It  was  not  optimal  but  an  acceptable 
alternative. 


2.4.1  The  interview  protocol 

The  following  describes  the  protocol  followed  by  the  interviewers  in 
each  session. 

1.  Each  Buyer/Price  Analyst  will  be  presented  with  a  set  of  cases 
one  at  a  time.  He  will  be  expected  to  complete  a  case  by 
generating  a  Price  Negotiation  Memorandum  (PNM)  documenting  the 
action  to  be  taken  with  respect  to  the  case.  When  a  case  is 
completed,  the  next  case  will  be  presented. 

2.  Each  case  package  consists  of  5  components: 


a.  Information  received  by  the  analyst  from  the  using  agency. 
These  consist  of  a  Request  for  Proposal  (RFP),  a 
government  estimate,  potential  sources,  the  transmittal 
letter,  and  the  statement  of  work  (not  all  these  items 
will  actually  be  present  in  all  cases). 

b.  Information  formally  received  from  contractors.  These 
include  the  actual  proposals/bids,  cover  letters, 
attachments  and  documentation  for  the  contractors  bid 
(such  as  a  price-list,  manufacturer's  catalogs,  etc.). 

c.  Information  received  by  the  analyst  from  the  contractors, 
using  agency,  and  other  sources  (such  as  Defense  Contract 
Audit  Agency  (DCAA),  General  Services  Administration 
(GSA),  various  technical  specialists,  etc.)  in  other  ways 
(such  as  telephone  conferences). 
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d.  The  Price  Negotiation  Memorandum,  abstract  of  bids,  or 
other  documentation  of  the  actual  action  taken  in  the  real 
case. 

e.  Worksheets  for  use  by  the  analyst.  In  this  study,  this 
will  be  WPAFB  Form  246  (AFLC  Form  1612).  This  form  is 
recommended  for  use  for  writing  up  PNMs  for  contracts  in 
the  range  $25,000  to  $500,000. 

3.  The  first  two  components  of  the  package  are  given  to  the 
analyst.  The  third  component  is  retained  by  the  interviewer  and 
the  information  therein  is  provided  to  the  analyst  if  the 
analyst  requests  it  while  working  on  the  case*  The  fourth 
component  is  for  documentation  purposes  only.  The  blank  PNM 
form  is  provided  as  a  convenience  (the  analyst  is  not  required 
to  use  it,  but  may  choose  to). 

4.  The  analyst  is  asked  to  verbalize  while  solving  the  case.  The 
interviewer's  primary  task  is  to  facilitate  this  process,  by 
asking  questions  if  necessary,  and  providing  information  from 
component  3  of  the  package,  if  requested.  The  interview  is 
taped  and  transcribed  for  later  analysis. 


2.4.2  General  questions 

The  interviewers  observed  the  buyer's /analyst's  behavior  and  made  notes 
to  answer  the  following  questions: 

1.  What  information  from  the  case  packet  is  used  or  requested  by 
the  analyst? 

2.  What  priority  does  the  analyst  assign  to  different  pieces  of 
information? 

3.  What  "outside"  information  does  the  analyst  request  or  use? 

4.  What  assumptions  does  the  analyst  make  about  information  that  is 
not  presented? 


5.  What  methods  and  procedures  does  Che  analyst  follow? 
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3.  ANALYSIS  OF  MODEL  (SOW  4.2.2) 

In  this  chapter,  we  identify  and  document  the  rules  and  experience  used 
by  USAF  price  analysts  in  analyzing  contractor  price  proposals.  The 
objective  of  the  interviews  and  briefings  was  to  define  the  knowledge  base 
required  for  carrying  out  the  pricing  function.  The  knowledge  base 
reflects  the  rules  and  experience  used  by  USAF  personnel  in  analyzing 
contractor  price  proposals.  Two  general  conclusions  were  reached  in  this 
regard.  First,  price  analysis  should  not  be  viewed  as  an  independent 
component  of  the  buyer's  task.  Decisions  made  and  information  gathered  at 
all  stages  of  the  process  have  a  direct  bearing  on  the  task.  The  buyer's 
task  is  to  establish  a  government  objective  with  which  to  compare  the 
proposed  price.  All  procurement  activities,  not  just  price  analysis,  are 
directed  toward  establishing  this  objective  price.  Second,  the  process 
appears  to  be  driven  by  the  Price  Negotiation  Memorandum  (PHM) .  The  PHM  is 
the  formal  documentation  of  the  buyer's  activities  in  consummating  a 
contract  at  a  "fair  and  reasonable"  price.  From  a  theoretical  stand  point, 
it  is  very  important  that  the  content  of  the  knowledge  base  be 
distinguished  from  the  procedures  followed  by  the  buyer/price  analyst. 

3.1  The  Task  Environment 

This  section  presents  the  activities  surveyed  and  discusses  the  specific 
rules  and  experience  needed  to  perform  price  analysis. 

3.1.1  Activities  Surveyed 

Price  analysis  as  carried  out  by  United  States  Air  Force  (USAF) 
procurement  activities,  is  surveyed  at  three  different  procurement  levels: 

1.  Air  Force  Systems  Command-Central  (AFSC  or  Systems  level 
procurement) . 

2.  Air  Force  Logistics  Command -Central  (AFLC  or  Logistics  level 
procurement ) . 


3.  Air  Force  Base  procurement  (Base  level  procurement) 
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These  activities  were  identified  by  consulting  with  OSAF  personnel  as 
representing  three  diverse  task  environments  (or  levels).  They  are 
compared  along  the  following  dimensions: 

a 

1.  Type  of  analysis  generally  undertaken. 

2.  Characteristics  of  procurements. 

3.  Type  of  accounting  systems  encountered. 

4.  Availability  of  data. 

5.  Expertise  of  personnel. 

6.  Utility  of  historical  data. 

7.  Online  computer  support. 

8.  Type  of  data  generally  used. 

9.  Availability  of  support  personnel. 

As  stated  previously,  the  survey  vas  carried  out  over  a  nine  month 
period  and  consisted  of  attending  briefings  by,  and  conducting  interviews 
with,  supervisory  and  staff  personnel  within  each  activity  as  well  as 
examining  relevant  official  publications  and  regulations. 

Figure  1  summarizes  the  task  environment  vithin  each  procurement  level 
along  the  dimensions  presented  above.  They  range  from  base  level 
procurement  which  is  generally  unsophisticated  and  under-supported  to 
Systems  level  procurement  which  is  highly  sophisticated  with  a  relatively 
high  degree  of  support. 

3.2  Specific  rules  and  experience  needed 

In  addition  to  the  general  statements  presented  in  Figure  1,  the 
following  specific  conditions  and  knowledge  have  been  identified  as 
necessary  for  effective  procurement. 

1.  Prior  experience  with  contractors  (e.g.,  prior  contract 
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Figure  1:  Dimensions  of  Procurement 
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performance,  reputation). 

2.  Prior  buys  (useful  at  the  base  level,  not  at  other  levels). 

3 .  Rules  given  by  regulation  or  operating  instructions : 


-  checklists  of  tasks  to  be  performed. 

-  file  and  data  base  organization. 

-  J041  routing  system  rules  and  implementation  procedures. 

4.  Idiosyncratic  rules  that  help  them  distinguish  among  technical 
terms . 

5.  Structure  of  procurement  activity  (this  differs  depending  on  the 
particular  type  of  procurement  and  type  of  item  being  procured). 

6.  Evaluation  of  overhead  (the  perception  that  a  high  overhead  is 
bad  is  simplistic  and  the  buyer/analyst  needs  to  be  aware  of 
this) . 

7 .  Error  detection  of  both  accounting  errors  and  possibly 
deliberate  errors.  (Simply  double-checking  line  items  can  be 
very  beneficial). 

8.  Using  a  variety  of  forms  (buyers  found  unfamiliar  forms  very 
disturbing) . 

9.  Knowledge  about  the  appropriate  production  and  manufacturing 
processes  and  who  the  experts  in  the  Air  Force  are  (who  to  call? 
what  information  is  needed?). 

10.  Degree  of  product  intensive  expertise  (analysts  at  the  logistics 
and  systems  level  often  specialize  in  particular  types  of  buys). 

11.  Speeding  up  the  procurement  processing  by  initiating  several 
subtasks  simultaneously. 

12.  Recognizing  and  identifying  sources  and  their  inter¬ 
relationships  (e.g.,  recognizing  that  one  quote  is  from  a  vendor 
and  another  one  from  the  manufacturer). 

13.  Scepticism  about  sole  source  justifications. 

14.  Objectivity  with  respect  to  the  government  estimate  (degree  of 
confidence  in  unrealistic  or  random  estimates  by  user). 
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4.  FEASIBILITY  (SOW  4.3) 

In  this  chapter  we  diacuaa  the  feasibility  of  developing  an  expert 
system  for  contract  pricing.  Based  on  information  gathered  in  the  prior 
phases,  we  define  the  knowledge  base  that  can  be  used  in  developing  an 
expert  computer  system. 

4.1  Knowledge  Base  of  the  design 

Expert  computer  systems  can  be  viewed  in  terms  of  their  decision  making 
capabilities  [2].  These  capabilities  range  from  an  integrated  intelligent 
system  to  a  basic  data  base  management  system.  A  complete  expert  system 
would  contain  the  necessary  databases  and  the  capabilities  to  perform  all 
procurement  functions  of  the  contracting  officer.  A  less  ambitious 
undertaking  would  concentrate  on  performing  one  or  more  of  the  several 
components  of  the  task,  with  the  ultimate  goal  being  the  integration  of 
these  components  into  a  complete  system.  To  do  this,  we  must  determine 
which  components  of  the  procurement  process  are  most  amenable  to 
performance  by  intelligent  computer  systems.  An  analysis  of  the  pricing 
task  is  undertaken  to  this  end.  The  following  sections  characterize  price 
analysis  in  the  Air  Force,  describe  a  system  architecture  for  a  viable 
system,  and  discuss  the  components  of  this  architecture. 

4.1.1  The  Price  Analysis  Task 

An  expert  computer  system  designed  to  support  price  analysis  must  be 
capable  of  functioning  within  the  Air  Force  procurement  envirooaent .  Three 
entities  are  involved  in  procurement:  the  using  agency,  the  contractor, 
and  the  contracting  officer  (also  called  the  buyer).  The  using  agency 
provides  a  set  of  technical  specifications  and  a  cost  estimate  (or  budget) 
of  the  proposed  procurement.  Based  on  the  requirements  set  forth  in  the 
solicitation,  the  contractor  submits  a  proposed  price  for  providing  the 
desired  item  or  service.  The  contracting  officer,  or  the  designated 
representative,  is  responsible  for  taking  the  user's  request,  soliciting 
proposals,  evaluating  the  proposals,  establishing  a  negotiating  position, 
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carrying  out  the  negotiations,  and  consummating  a  contract.  Throughout 
this  process,  the  buyer  may  need  to  obtain  in-depth  technical,  pricing,  or 
legal  knowledge  from  specialists  in  these  areas.  The  following  discussion 
presents  the  procurement  task. 

For  the  purpose  of  this  research,  we  considered  there  to  be  six  phases 
in  the  procurement  process  [171: 

1.  Evaluate  Purchase  Request  (or  a  MIPS  from  another  am  of  the 
Department  of  Defense). 

2.  Issue  solicitation. 

3.  Receive  proposals  or  bida. 

4.  Evaluate  proposals. 

3.  Issue  a  contract. 

6.  Administer  the  contract. 

In  the  following  sections,  we  discuss  each  phase  in  terms  of  its 
relation  to  the  price  analysis  task  and  the  knowledge  base  needed  to  design 
a  prototype  intelligent  manual. 

4.1.1 .1  Evaluate  Purchase  Request  or  MIPS 

The  following  information  is  provided  by  the  using  agency  generating  the 
purchase  request  and  is  generally  checked  by  the  buyer: 

1.  Priority  class  of  procurement. 

2.  Government  estimate  (according  to  the  using  agency). 

3.  Delivery  schedule  or  performance  dates  (proposed  by  the  using 
agency) . 

4.  Funds  authorization  (provided  by  the  funding  agency). 

5.  User  requirements  (generated  by  the  using  agency  experts). 

6.  Sole  source  justification  (if  the  using  agency  believes  that  the 
item  should  be  procured  from  a  single  source). 
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In  this  phase,  the  buyer's  primary  task  is  to  check  that  the  requisite 
information  has  been  supplied,  verify  it  with  the  user  or  a  third  party, 
and  question  possible  excesses.  The  most  probable  areas  for  excesses  that 
can  influence  the  final  price  are  the  priority  class,  a  sole  source 
procurement,  the  specifications,  and  the  delivery  schedule  and  performance 
requirements. 

4.1 .1.2  Issue  Solicitations 

the  following  represent  the  major  components  of  the  solicitation  phase: 


1.  Generate  formal  specifications. 

2.  Determine  a  contract  type. 

3 .  Identify  and  verify  funds  authorization. 

4.  Identify  contract  line  items. 

5.  Determine  appropriate  contract  clauses. 

6.  Determine  and  contact  possible  sources. 

7.  Determine  the  kind  of  solicitation. 


Some  of  this  information  may  already  have  been  obtained.  For  example, 
the  specifications  for  the  procurement  and  the  line  items  in  the  contract 
will  have  been  provided  by  the  user  as  part  of  the  purchase  request.  The 
buyer  must  then  perform  the  other  components  of  the  above  phase.  For 
example,  the  buyer  must  choose  between  s  fixed-price  type  of  contract  and  a 
cost-reimbursement  contract.  The  buyer  must  slso  identify  possible  sources 
of  supply,  determine  the  contract  clauses  to  include  and  the  kind  of 
solicitation  to  undertake. 

Contract  types  include: 


1.  Firm  fixed  price  (FFF). 

2.  Fixed  price  incentive-firm  target. 

3.  Fixed  price  incentive-successive  targets. 
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4.  Fixed  price  with  redetermination. 

5.  Coat  plus  incentive  fee. 

6.  Coat  plus  award  fee. 

7.  Coat  plua  a  fixed  fee. 

8.  Cost  contract. 

9.  Cost  sharing. 

10.  Time  and  materials. 

11.  Labor  hours. 

The  clauses  to  be  included  in  the  contract  are  selected  from  a  list  of 
general  clauses  organized  under  the  following  sections,  as  well  as  from 
subclauses  appropriate  to  the  procurement. 

A.  Contract  Form  Information. 

B.  Supplies /Services  and  Prices. 

C.  Deacription/Specification. 

D.  Packaging  and  Marking. 

E.  Inspection  and  Acceptance. 

F.  Deliveries  or  Performance. 

G.  Contract  Administration  Data. 

H.  Special  Provisions. 

I.  General  Provisions. 

J.  List  of  Documents,  Exhibits  and  Other  Attachments. 

K.  Representations,  Certifications,  and  other  Statements  of 
Offeror. 

L.  Instructions  and  Conditions  and  Notices  to  Offerors. 

M.  Evaluation  Factors  for  Award. 
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The  user  provides  possible  sources  of  supply;  however,  the  buyer 
supplements  this  using  a  history  of  prior  buys  and  a  list  of  contractors 
maintained  by  the  procurement  office.  The  buyer  must  also  choose  betveen 
issuing  a  Request  For  Proposals  (RFP)  or  an  Invitation  for  Bids  (IFB). 

This  decision  is  based  on  the  complexity  and  commonality  of  the  item  to  be 
procured,  the  buyer's  perception  of  the  competitiveness  of  the  situation 
and  the  local  availability  of  suppliers.  The  item  itself  can  influence  the 
procedures  followed  in  the  subsequent  phases  of  the  procurement. 

4.1 .1.3  Receive  Proposals  or  Bids 

Upon  receipt  of  the  proposals /bids,  the  buyer's  task  is  to  extract  the 
following  information  from  that  submitted  by  the  contractors. 

1.  Total  proposed  price. 

2.  Price  breakdown  by  line  item,  part  number,  option,  etc. 

3.  Exceptions  to  technical  specifications. 

4.  Exceptions  to  delivery  schedules. 

3.  Exceptions  to  performance  requirements. 

The  information  provided  by  contractors  can  be  used  to  generate  the 
government  baseline  for  negotiations.  For  example,  the  buyer  can  determine 
that  an  user-provided  estimate  for  the  unit  price  of  a  certain  line-item  is 
inappropriate  and  not  supported  by  any  contractor.  Thus  contractor 
provided  data  plays  a  crucial  role  in  developing  the  government  objective. 

The  buyer  is  also  responsible  for  acquiring  other  contractor  information 
relevant  for  making  comparisons  between  contractors  and  with  the  government 
objective.  They  are: 

1.  History  of  performance  on  past  contracts. 

2.  Availability  of  financial  information  on  the  contractor. 

3.  Special  of  contractor  designation  (minority-ovned  or  small 
business,  etc.),  if  any. 


4.  Demographic  information. 

5.  Government  contacts  within  the  company. 

6.  Other  government  contracts  with  the  company 
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4.1 .1.4  Evaluating  the  Contract 

In  the  contract  evaluation  phase,  the  buyer's  primary  tasks  are  that  of 
determining  a  government  objective,  determining  the  most  advantageous 
proposal  and  negotiating  a  contract.  There  are  four  phases  in  this  task. 

1.  Objective  development. 

2.  Negotiation  planning  and  review. 

3.  Negotiation. 

4.  Final  contract. 

Objective  development  is  the  process  performed  by  the  buyer  in  deriving 
a  government  objective  for  the  total  price  of  the  contract  or  for  its  line 
items.  The  objective  is  primarily  a  price  that  the  governeent  perceives  to 
be  "fair  and  reasonable”,  but  technical  specifications,  quantity  and 
delivery  schedules  are  other  important  factors.  The  activities  listed 
below  are  carried  out  during  objective  development. 

1.  Verifying  user  requirements. 

2.  Fact  finding. 

3 .  Price  Comparisons . 

4.  Exception  comparisons. 

5.  Government  objective  comparisons. 

In  the  previous  phases,  the  major  activities  are  information-gathering 
tasks.  The  objective  development  phase  requires  extensive  information 
processing.  The  buyer  must  verify  that  the  proposals  meet  the  technical 
requirements  and  the  delivery  schedules  proposed  by  the  user.  These  are 
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routine  tasks  if  proper  procedures  are  followed  prior  to  solicitation, 
unless  there  is  an  indication  of  possible  problems  or  cost  savings.  When 
contractors  take  exception  to  solicitation  requirements  the  buyer  judges 
whether  further  investigation  would  be  beneficial. 

Fact-finding  is  the  next  phase  performed  by  the  buyer.  It  consists  of 
the  following  activities  and  decisions: 

1 .  Eliminate  sources . 

2.  Evaluate  prior  buys. 

3.  Request  and  receive  field  reports. 

4.  Identify  facilities  to  be  provided  by  the  government,  if  any. 

3.  Identify  relevant  outside  influences  on  the  procurement,  if  any. 

6.  Evaluate  time  pressures,  if  any. 

7.  Document  activities  performed  as  part  of  objective  development. 


Sources  may  be  eliminated  if  proposals  are  unacceptable  under  any  of  the 
following  criteria: 

1.  Technical  reasons  (whether  the  contractor's  specifications 
agrees  with  the  user's  proposed  specifications). 

2.  Legal  reasons  (if  there  are  legal  reasons  why  a  contractor  stay 
not  be  considered). 

3.  Debarred  (if  the  contractor  is  not  eligible  for  government 
contracts) . 

4.  Delivery  schedule  (if  the  contractor's  proposed  delivery 
schedules  are  unacceptable  to  the  user). 

5.  Unreasonable  bid  (if  all  offers  are  deemed  "too  big”). 

6.  Evaluation  of  competition  (if  the  situation  is  deemed  to  be  non¬ 
competitive)  . 


When  establishing  a  government  objective,  the  buyer  accesses  prior 
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procurement  history  of  several  types. 

1.  Buys  of  the  same  item. 

2.  Buys  of  similar  items. 

3.  Follow-on  buys. 

The  identification  of  "similar”  (and  "same")  buys  is  based  on  similarity 
(or  identity)  of  specifications  maintained  in  a  database  of  prior 
contracts . 


The  buyer  must  request  and  analyze  field  reports  from  specialists  such 
as  price  analysts,  DCAA  auditors,  and  technical  experts.  The  buyer  must 
also  determine  what  industrial  facilities  are  to  be  provided  by,  or  are 
available  from,  the  government.  The  effect  of  outside  influences  such  as 
the  changing  prices  for  certain  raw  materials,  the  availability  of  required 
skilled  labor,  and  sole  source  requirements  must  be  identified  and 
evaluated  as  to  the  extent  of  their  impact  on  price.  Time  pressures 
arising  from  user  specified  delivery  schedules  or  period  of  performance 
must  be  evaluated.  Finally,  the  fact-finding  process  must  be  documented  as 
to  source,  date,  medium  of  communication  and  consequence. 

One  result  of  fact  finding  is  to  select  one  of  the  following  procedures 
as  appropriate  for  establishing  a  government  objective: 


1.  Competitive  Pricing. 

2.  Published  Pricing  (based  on  Catalog,  Market  or  Regulated 
prices ) . 

3.  Secondary  comparisons  (such  as  prior  quotes  or  formulae  for 
developing  an  estimate). 

4.  Cost  Analysis . 


The  result  of  fact-finding  and  the  evaluation  of  the  proposals  submitted 
is  a  government  objective  established  by  the  buyer.  Once  the  government 
objective  is  established,  the  proposal/bids  are  evaluated  in  terms  of  this 
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objective  price,  the  prices  proposed  by  the  other  contractors,  and  the 
exceptions  taken  by  the  contractors  to  the  solicitation  specifications. 

Price  analysis  is  generally  defined  as  the  task  of  determining  a  fair 
and  reasonable  price  for  the  buy.  In  the  simplest  situation,  it  may  only 
involve  comparing  two  numbers  and  choosing  the  smallest.  It  may  require 
cost  analysis  —  that  is  constructing  a  price  based  on  full  costs  generated 
from  engineering  specifications.  The  information  gathered  during  fact 
finding  indicates  the  appropriate  type  of  analysis  and  provides  useful  data 
for  performing  the  analysis.  Some  of  the  information  is  necessarily 
provided  by  specialists  who  can  be  accessed  by  the  price  analyst.  For 
example,  field  personnel  can  provide  information  as  to  the  level  of 
compliance  with  Cost  Accounting  Standards  Board  (CASB)  requirements  and 
technical  specialists  can  provide  evaluations  of  proposed  work  processes, 
material,  and  labor  specifications.  One  system  design  problem  is  choosing 
which  tasks  to  include  as  part  of  the  price  analysis  function  and  which  to 
treat  as  specialty  areas  to  be  called  by  the  price  analysis  system  when 
needed  [10]. 

During  the  negotiation  planning  and  review  phase,  the  major  activity  is 
to  evaluate,  document,  and  review  the  government  position  with  respect  to 
the  following  issues: 

1.  Justification  for  the  government  position. 

2.  Assumptions  made  by  the  price  analyst  as  well  as  those  made  by 
the  contractor. 

3.  Strengths  and  weaknesses  of  the  government  position. 

4.  Strengths  and  weaknesses  of  the  contractor's  proposal. 

5.  Relevant  range  of  estimates. 

The  following  activities  are  undertaken  as  part  of  the  negotiation 


process : 
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1.  Selection  of  contractor^ )  with  whom  to  negotiate. 

2.  Resolution  of  identified  differences  in  facts. 

3.  Resolution  of  identified  differences  in  judgment. 

4.  Documentation  of  the  process  and  the  final  position  agreed  upon. 

The  final  contract  is  drawn  up  based  upon  the  agreements  reached  during 
the  negotiations . 

4.2  Prototype  Development  (SOW  4.3.2) 

In  this  section,  we  present  the  proposed  architecture  for  an  intelligent 
manual  as  well  as  an  illustrative  module  of  the  resulting  prototype  system. 

A  complete  expert  system  would  have  the  following: 

1.  Decision  making  capabilities. 

2.  Decision  support  capabilities. 

3.  Functions  to  specify  and  support  specific  company  or  company 
type  models . 

4.  Oser-friendly  interfaces  for  computer-based  systems. 

3.  Standardization  within  companies  across  contracts  and  across 
companies  for  the  same  contract. 

6.  Mechanisms  to  identify  cost-estimating  relationships  and  to 
catalog  and  retrieve  Forward  Pricing  Agreements  (FPRAs). 

7.  Access  to  available  field  data. 

For  the  task  evaluated,  it  would  be  very  costly  to  implement  and  support 
this  type  of  system.  For  example,  such  a  system  would  rely  on  a  large 
database  of  contracts  and  work  breakdown  information  that  is  generally  not 
available.  We  propose  that  the  best  initial  approach  is  to  implement  an 
intelligent  manual  expert  system  and  to  begin  to  construct  more 
"intelligent"  subsystems,  or  "specialists",  that  can  eventually  be  joined 
to  provide  a  total  system  [10,  3,  4],  This  is  a  realistic  alternative 
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given  current  technology  and  the  database  requirements.  An  intelligent 
manual  can  be  implemented  fairly  quickly,  will  not  require  a  great  deal  of 
personnel  retraining  to  implement,  and  does  not  require  an  extensive 
historical  data  base.  The  system  architecture  and  a  prototype  of  an 
intelligent  manual  are  presented  below. 

4.2.1  Architecture 

The  architecture  of  the  intelligent  manual  has  four  major  components 
shown  in  Figure  2.  They  are  the  Task  Support  System,  the  Task  Guidance 
System,  the  Task  Action  System,  and  the  External  Interface  System. 

The  Task  Support  System  supports  the  basic  task  by  providing  an 
appropriate  organization  for  the  overall  task.  The  organization  is 
obtained  by  determining  how  experts'  perform  the  task  is  designed  to 
support  performance  at  that  level. 

The  Guidance  System  provides  explanation  and  guidance  for  performing  the 
basic  task.  The  structure  of  this  component  parallels  the  structure  of  the 
Task  Support  System  so  that  a  novice,  on  requesting  help,  is  given  the 
appropriate  explanation  and  is  guided  to  perform  the  appropriate 
activities. 

The  Task  Action  System  consists  of  the  programs  and  tools  (such  as 
statistical  analysis  programs,  text  editors,  learning  curve  programs, 
databases,  etc.)  that  are  necessary  for  performing  the  analysis  needed  in 
the  Task  Support  system.  These  programs  can  be  invoked  from  the  Task 
Support  system  (as  well  as  the  Guidance  system);  the  input  data  required  by 
these  programs  is  provided  via  the  Task  Support  or  Guidance  systems;  and 
the  generated  output  data  is  accessible  from  the  Task  Support  or  Guidance 
systems. 

The  External  Interface  System  uses  the  results  of  the  users  interaction 
with  the  above  three  systema  to  generate  appropriate  output.  For  example, 
this  system  may  generate  a  structured  output  that  documents  the  user's 
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decision  or  an  audit  trail  report  that  specifiea  the  actiona  performed  by 
the  uaer  and  the  poaaible  reaaona  for  theae  actiona.  Many  different  kinda 
of  output  are  poaaible  and  all  of  them  muat  be  baaed  on  information 

gathered  from  the  user's  interaction  with  the  other  three  componenta. 

• 

4.2.1 .1  Brief  Introduction  to  ZOG 

The  intelligent  manual  prototype  is  conatructed  on  top  of  the  ZOG 
information  ayatem.  ZOG  ia  a  system  for  human-computer  conmunication, 
characterized  by  a  rapid-response,  menu-selection  interface  to  a  large, 
network-structured,  active  database  [14,  13].  ZOG  has  been  applied  in  a 
number  of  different  data  base  and  project  management 

situations  [5,  6,  9,  12].  A  ZOG-user  moves  from  frame  to  frame  using  the 
computer  terminal  to  view  the  contents  of  one  frame  at  a  time.  Frames  are 
designed  to  be  displayed  on  a  single  screen.  By  convention,  every  frame 
has  a  one-line  title  at  the  top  of  the  screen,  a  few  lines  of  text  below 
the  title,  a  set  of  numbered  (or  lettered)  menu  itasu  of  text  called 
"selections",  and  a  line  of  ZOG  commands  called  "global  pads”  at  the  bottom 
of  the  screen. 

Frames  are  interconnected  by  the  menu  items.  When  the  user  selects  an  * 
item  (by  typing  its  number  or  letter,  or  by  touching  the  screen  location  of 
the  item),  ZOG  ’Vnoves"  the  user  to  the  frame  "pointed  to"  by  the  selection. 
This  new  frame  is  displayed  on  the  screen,  replacing  the  frame  from  which 
the  selection  was  made.  The  new  frame  will  have  the  same  general  format; 
it  will  usually  contain  new  information  and  further  selections  that  lead  to 
more  detailed  information.  Basically,  the  frame  network  is  a  hierarchical 
information  structure  with  extensive  cross-referencing  as  well  as 
mechanisms  for  moving  directly  from  frames  deep  down  in  the  hierarchy  to 
frames  much  higher  up.  The  network  of  frames  is  often  termed  a  ’^OGnet." 

Selections  and  frames  may  have  associated  "actions"  that  activate 
programs  (or  other  entities)  on  behalf  of  the  user.  These  actions  are 
executed  when  the  frame  is  displayed  or  when  a  selection  is  made  by  the 
user.  These  actions  implement  the  connecting  link  between  the  Task  Support 
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system  and  the  Task  Action  Support  system  of  the  intelligent  manual. 
Figure  3  shows  an  example  ZOG  frame. 
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Figure  3:  An  Example  ZOG  Frame  Estimatel 

The  ZOG  system  is  primarily  used  to  organize  the  Task  Support  System  and 
the  Task  Guidance  System.  Functions  provided  by  the  Task  Action  System  and 
the  External  Interface  System  can  be  controlled  from  ZOG  via  the  selection 
mechanism. 

4.2.2  Structure  of  the  Task  Support  System 

The  Task  Support  System  is  organized  as  a  hierarchy.  Every  node  in  the 
hierarchy  is  implemented  as  a  frame  in  ZOG.  The  hierarchy  is  defined  by 
the  hierarchical  organization  of  the  tasks  that  the  buyer  must  perform 
during  procurement.  This  organization  has  been  discussed  earlier  and  is 
shown  briefly  in  Figure  4. 
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4.2 .2.1  Implementation  of  the  Prototype 

There  are  three  basic  types  of  frames  in  the  ZOG  prototype.  They  are: 

1.  Node  frames  that  maintain  information  about  some  task  or  sub¬ 
task  of  procurement . 

2.  Decision/Selection  frames  that  maintain  information  about 
decisions  made  during  procurement . 

3.  Database  frames  that  contain  pricing  and  other  information. 

Node  frames  allow  the  user  to  focus  on  a  subtask,  as  well  as  provide 
context  to  organize  the  task.  The  Decision/ Selection  frames  provide  the 
structure  for  the  user  to  make  decisions  by  appropriate  selections  at  the 
frame.  They  contain  local  pads  that  enable  the  user  to  make  the  selections 
interactively.  They  also  provide  the  user  information  on  making  the 
decision  as  well  as  the  ability  to  postpone  making  a  final  decision  or  to 
make  partial  decisions.  The  Database  frames  provide  the  user  access  to 
data  on  contracts.  They  contain  mechanisms  that  allow  the  user  to  fill  in 
and  edit  data  as  well.  The  typical  database  frame  is  organized  as  a  two- 
dimensional  table,  though  other  structures  (lists,  etc.)  are  also  possible. 

Every  frame  has  an  associated  "Help"  local  pad.  In  the  case  of  Node 
frames,  these  connect  to  parallel  frames  of  the  Task  Guidance  System.  For 
Decision/Selection  frames,  they  connect  to  frames  in  the  Task  Guidance 
System  that  discuss  the  decisions  to  be  made  and  provide  help  in  making  the 
decisions.  For  Database  frames,  they  lead  to  frames  that  explain  the 
functions  available  as  well  as  explain  the  data  being  displayed. 

Figure  5  shows  a  Node  frame.  The  options  in  the  node  frame  lays  out  the 
subtasks  that  are  needed  to  perform  the  task  identified  by  the  frame.  The 
text  of  the  frame  is  (or  should  be)  minimal,  if  any.  In  the  frame  shown  in 
Figure  3,  for  example,  the  text  simply  classifies  all  the  sub-tasks  into 
two  groups . 


Figure  6  shows  a  Decision/Selection  frame.  This  frame  allows  the  user 
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Figure  5:  A  Node  frame  —  The  Solicitation  task 


has  to  determine  what  items  being  procured  are  likely  to  be  affected  by 
time  pressures.  The  items  and  the  contract  option  that  they  occur  under 
are  the  rows  and  columns  of  the  matrix.  The  user  can  select  from  the  set 
of  local  pads.  For  example,  if  the  user  decides  that  Item  1  is  not 
affected  by  time  pressures  at  all,  he  selects  the  E.  Row  not  affected  local 
pad  and  then  types  '1'  to  specify  the  row. 

Figure  7  shows  a  Database  frame. 

This  frame  shows  the  total  price  proposals  by  different  contractors  on 
the  main  and  optional  parts  of  the  solicitation.  It  is  organized  as  a 
matrix.  There  may  be  more  options  than  can  be  shown;  similarly,  there  may 
be  more  than  the  eight  proposals  shown.  In  this  case,  the  matrix  is 
sxtended  over  many  frames  and  every  frame  only  displays  a  "window"  into  the 
matrix.  There  are  ZOG  local  pads  (F.  First.  L.  Last.  >.  More.  <■  Prev.  and 
others)  that  provide  mechanisms  for  moving  to  different  windows  (i.e., 
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Figure  6:  A  Decision/Selection  frame  —  Selecting  Contract  Type 


frames).  There  are  also  local  pads  that  allow  entry  of  new  information  and 
deletion  or  modification  of  old  information,  so  that  the  same  frame  can  be 
used  to  create  the  matrix  initially. 

A  database  frame  is  really  a  point  of  entry  for  the  user  into  a 
database.  For  particular  databases,  there  may  be  better  mechanisms  for 
access,  use,  and  modification  of  the  database  than  the  ZOG  frame  structure. 
As  explained  earlier  (using  the  Task  Action  System),  ZOG  can  make  system 
improvements  available  to  the  user.  So  a  mismatch  between  ZOG  structure 
and  a  database  structure  should  not  be  an  insuperable  barrier. 

4.2.3  Structure  of  the  Task  Guidance  System 

The  ZOG-based  pricing  guide  is  intended  to  guide  the  user  through  a 
series  of  questions  that  must  be  raised  in  procurement  situations.  If  the 
user  cannot  answer  a  particular  question,  the  system  breaks  the  issue  down 
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Figure  7:  A  Database  frame  —  Proposal  Comparisons 


into  more  basic  issues.  For  example,  the  question  "Is  this  a  competitive 
situation?"  can  be  broken  dovn  into  such  questions  aa : 

1.  Are  there  tvo  or  more  independent  contracta? 

2.  Do  each  of  the  contractora  satisfy  the  solicitation  criteria? 

3.  Is  the  procurement  object  generally  available  to  the  public? 

In  the  firat  level  of  the  system,  the  user  will  be  guided  to  ansvering 
questions  by  breaking  them  dovn  into  simpler  questions.  This  procedure 
will  not  necessarily  aid  a  complete  novice.  The  novice  (and  experts,  too) 
may  find  examples  and  definitions  of  terms  useful.  The  second  level  of  the 
system  provides  this  kind  of  support.  Examples  and  elaborations  of 
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questions  and  the  kinds  of  answers  expected  by  the  system  are  provided  to 
the  user.  This  may  still  not  suffice.  At  the  third  level  (currently  the 
lowest  level  that  has  been  defined),  the  user  will  be  directed  to  a 
"learning  net”.  This  is  a  piece  of  the  ZOGnet  that  presents  the 
information  needed  by  the  user  in  a  textbook/tutorial  style.  In  this 
presentation,  we  only  present  the  design  of  the  first  level  ZOGnet  (the 
guidance  system) . 

Figure  8  displays  the  complete  structure  of  the  pricing  guide.  We  have 
chosen  a  somewhat  abbreviated  notation  to  indicate  some  of  the  issues, 
questions,  and  sub-questions  that  must  be  raised  by  the  user. 

In  the  first  level,  every  frame  contains  four  components:  an  Information 
component  that  provides  the  user  some  information;  an  Action  component  that 
instructs  the  user  to  do  something;  a  Decision  component  that  poses  a 
question  to  the  user  that  must  be  answered  with  reference  to  the 
contract(s);  and,  an  Potion  component  that  provides  the  user  with  a  set  of 
possible  answers  to  the  question. 

The  Decision  and  Option  components  determine  the  structure  of  the 
network.  Most  questions  will  have  "Yes /No"  answers,  though  others  may  have 
slightly  more  complex  answers.  In  any  case,  the  questions  are  formulated 
so  that  they  can  be  answered  from  a  small  menu.  Figure  9  shows  a  question 
that  the  buyer  is  confronted  with  by  the  system:  "Is  each  offer  priced  and 
responsive  to  the  requirements  of  the  solicitation"?  The  answers  can  be 
Yes  or  No.  Such  a  frame  is  called  a  Decision-Option  frame  —  there  are  no 
Actions.  The  information  presented  in  the  frame  directly  explains  the 
question.  If  the  answer  is  "yes"  or  "no"  (selects  menu  items  1.  or  2, 
respectively),  the  buyer  will  be  taken  to  a  Review  frame  (see  Figure  10  for 
an  example).  This  gives  the  buyer  the  opportunity  to  review  the  decision 
(by  reading  the  information  presented)  before  taking  an  action.  In  the 
case  that  the  buyer  is  not  sure,  the  S.  SPECIFY  local  pad  will  lead  to  a 
sequence  of  Decision-Option  frames  that  elaborate  on  the  question.  These 
will  ultimately  terminate  in  some  other  Review  frame  (or  frames).  If  the 


System  for  Price  Analysis 


Contract  Comparison  Phase 
Method  co  use? 


The  Design  of  an  Expert 


buyer  is  totally  confused  as  a  result  of  unfamiliarity  with  the  domain,  the 
E,_  EXAMPLE  local  pad  will  lead  to  an  example,  while  the  L.  LEARN  local  pad 
will  lead  the  buyer  to  the  learning  ZOGnet. 

A  Review  frame  typically  presents  the  buyer  with  a  summary  of  the 
reasons  for  making  the  decision  and  requests  confirmation.  The  buyer  then 
has  the  opportunity  to  retract.  In  the  example  shown  (Figure  10),  the 
buyer  may  realize  that  two  proposals  do  not  come  from  independent 
contractors  (this  requirement  may  have  been  ignored  by  mistake  earlier). 

The  buyer  can  select  1.  Continue  to  go  on  to  perform  the  action  associated 
with  the  positive  decision  or  select  7 .  Drirmiw»nt:  to  perform  the  action 
associated  with  the  negative  decision,  or  select  S.  SPECIFY  to  postpone 
making  a  definite  decision. 

Once  a  decision  has  been  made  and  reviewed,  the  guidance  system  also 
provides  mechanisms  for  implementing  the  action  to  be  taken  using  Action 
frames.  An  example  is  shown  in  Figure  11.  The  frame  text  explains  the 
action(s)  and  instructs  the  buyer  in  performing  the  action.  Selecting  1 . 
Continue  allows  the  buyer  to  perform  the  action  while  2.  Return  allows  the 
buyer  to  postpone  the  action  (and  possibly  retract  the  decision  to  perform 
it).  If  the  buyer  is  uncertain  of  how  to  perform  it,  the  S.  SPECIFY  local 
pad  will  provide  further  detail  and  explanation. 

The  above  example  of  a  ZOG  system  is  not  a  problem-solving  or  a  problem- 
analysis  system.  It  shows  the  underlying  model  of  the  domain  that  would 
help  a  user  perform  the  task  of  price  analyst  without  necessarily  having 
the  expertise  or  the  training.  However,  it  also  performs  another  important 
function.  It  identifies  the  questions  that  an  intelligent  system  must 
answer  and  is  useful  in  identifying  the  information  necessary  to  answer 
these  questions.  Some  of  this  information  must  be  obtained  from  the  source 
documentation  such  as  a  purchase  requisition;  some  of  this  information  must 
be  obtained  from  knowledge  of  the  regional  and  national  economy;  other 
information  is  unique  to  the  contractor  or  contractors.  All  of  this 
knowledge  must  be  accessible  to  the  intelligent  manual  in  an  appropriate 
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Each  solicitation  has  specific  requirements  about  the  type  of 
contract  that  is  acceptable  (e.g.,  firm  fixed  price,  cost-plus, 
etc.)  and  a  requirement  as  to  how  much  pricing  data  must  be 
given.  You  must  determine  if  each  offer  is  priced  and 
responsive  to  these  requirements . 


IS  EACH  OFFER.  PRICED  AMD  RESPONSIVE  TO  THE  REQUIREMENTS  OF  THE 

SOLICITATION? 


1.  Yes 

2.  No 


S.  SPECIFY  E.  EXAMPLE  L.  LEARN 

edit  help  back  next  mark  return  zog  display  user  goto  find  info 


Figure  9:  A  Decision-Option  frame 


representation. 


4.2.4  Task  Action  and  External  Interface  systems 

The  Task  Action  System  and  the  External  Interface  System  are  necessary 
components  of  the  intelligent  manual  architecture.  However,  the  design  of 
these  systems  is  largely  determined  by  task  considerations.  For  example, 
the  system  must  generate  solicitations  to  send  to  contractors.  The  buyer's 
decisions  about  the  content  and  form  of  the  solicitation  are  determined  by 
the  Task  Support  system.  The  External  Interface  system  can  generate  a 
complete  solicitation  and  send  it  off  automatically.  The  design  of  the 
solicitation  at  this  level  (legal  document,  on  paper,  etc.)  is  independent 
of  the  buyer's  problem-solving  or  information-gathering  activity. 

Similarly,  the  PNM  can  be  generated  automatically  from  the  documented 
decisions  of  the  buyer. 


The  Task  Action  system  provides  the  buyer  with  a  variety  of  utilities 
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iBy  claiming  that  a  competitive  situation  exists,  you  are  claiming  I 

I  that:  '  I 

I  I 

I  (1)  At  least  2  responsible  offerors  responded  to  the  solicitation  I 

!  and  passed  the  Contract  Selection  Phase.  I 

I  (2)  The  offerors  independently  contended  for  the  contract.  I 

I  I 

I  I 

!  I 

I  IS  THE  SITUATION  COMPETITIVE?  I 

I  I 

I  I 

!  1.  Continue  I 

I  I 

I  2.  Document  (not  a  competitive  situation)  ! 

I  i 

I  I 

I  I 

I  S.  SPECIFY  E.  EXAMPLE  L.  LEARN  I 

I  I 

I  edit  help  back  next  mark  return  zog  display  user  goto  find  info  I 

| - [ 

Figure  10:  A  Review  frame 

For  example,  the  Task  Action  system  may  contain  programs  for  plotting 
learning  curves.  The  data  for  the  programs  may  be  obtained  from  the  buyer 
via  ZOG  or  may  be  available  as  online  data,  or  may  already  be  a  part  of  ZOG 
in  Database  frames.  The  output  of  the  program  should  be  accessible  to  the 
buyer  via  ZOG.  The  Task  Action  system  therefore  consists  of  three  kinds  of 
programs : 

1.  Programs  that  know  how  to  interface  with  the  ZOG  system  and  the 
ZOGnet  structure. 

2.  Programs  that  ZOG  can  control  (via  terminal-mode  commands). 

3.  Programs  that  ZOG  can  communicate  with  and  vice-versa. 

For  example,  COPPER  IMPACT  programs  can  be  incorporated  into  the  system. 
Since  these  programs  have  not  been  designed  with  ZOG  in  mind,  they  cannot 
be  used  in  either  the  first  or  second  categories.  However,  the  Task 
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Those  offers  Chec  fail  Co  aeet  Che  cechaical  specifications 
of  Che  solicicadoa  must  eicher  be  revised  and  resubmitted  or 
eliaiaaced  from  coasideraCioa. 


I  ELIMINATE  THOSE  OFFERS  THAT  DO  NOT  MEET  THE  TECHNICAL  I 

I  SPECIFICATIONS  OF  THE  SOLICITATION  AND  CANNOT  BE  REVISED  I 

I  AND  RESUBMITTED  I 

I  I 

I  I 

I  1 .  Concinue  I 

I  I 

I  2.  Re C urn  I 

I  I 

I  I 

I  I 

I  S.  SPECIFY  E.  EXAMPLE  L.  LEARN  I 

I  I 

I  edic  help  back  next  nark  return  zog  display  user  goto  find  info  I 

| - 1 

Figure  11:  An  ACTION  frame 

Support  System  and  Task  Guidance  Systems  can  be  designed  with  knowledge  of 
COPPER  IMPACT  programs  and  can  control  it  via  4  simulated  terminal 
interface.  The  output  of  these  programs  can  be  analyzed  and  incorporated 
into  Che  database  and  accessed  via  a  ZOG  interface  to  the  database.  Over  a 
period  of  time,  these  programs  can  be  modified  so  that  they  communicate 
effectively  with  the  support  and  guidance  systems. 

Once  Che  support  and  guidance  systems  are  established,  programs  can  be 
written  that  understand  the  ZOGnet  structure  and  can  get  their  input  data 
from  the  net  and  incorporate  their  output  directly  into  the  net.  These 
programs  would  be  closely  integrated  into  the  intelligent  manual. 

4.3  Conclusion 

The  major  benefits  of  intelligent  manual  expert  systems  for  procurement 
decision  making  in  the  Air  Force  are: 
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1.  A  reduction  of  the  skill  level  necessary  to  perform  routine 
procurement  tasks . 

2.  Automatic  documentation  of  the  steps  taken  in  performing  the 
task. 


The  types  of  assistance  that  could  be  provided  within  the  intelligent 
manual  alternative  are: 


1.  Guide  price  analysis  by  providing  question  sequences  and 
elaborating  and  explaining  issues  surrounding  the  questions. 

2.  Provide  on-line  general  and  specific  indices. 

3.  Make  available  data  bases  of  historical  contract  data. 

4.  Make  available  general  models  of  contractor  types. 

5.  Specify  general  parameters  for  analysis. 

6.  Provide  cost  modeling  of  the  contractor's  estimation  systems. 

7.  Provide  guidance  in  establishing  parametric  and  predetermined 
rates . 

8.  Provide  access  to  previously  established  parametric  and 
predetermined  rates . 


The  greatest  benefit  of  implementing  an  intelligent  manual  would  be  in 
environments  where  little  expertise  or  support  is  available  (Base  level 
procurement  is  clearly  such  an  environment).  Two  major  objectives  could  be 
achieved.  First,  the  user  would  be  provided  with  price  analysis  assistance 
which  is  not  currently  available.  Instead  of  having  to  rely  on  expert 
support  that  is  not  always  available,  the  user  will  be  able  to  perform  the 
task  directly.  The  system  will  provide  structure  as  well  as  data  required 
for  the  task.  The  skill  level  required  to  do  price  analysis  would  be 
reduced.  Second,  the  audit  and  evaluation  processes  will  be  expedited. 

The  system  is  designed  to  take  the  responses  provided  by  the  user  and 
construct  the  required  verification  documentation  as  well  as  to  provide  an 
audit  trail  of  the  steps  taken.  Also,  necessary  indices  as  well  as 
pertinent  procedures  and  regulations  would  be  readily  accessible. 


System  for  Price  Analysis 

Analytical  tools  and  spread  sheet  capabilities  would  be  made  viable. 

In  tbe  long-term,  problem-solving  expert  systems  can  be  constructed  to 
perform  more  of  the  pricing  task  and  to  provide  more  aid  to  other  levels  of 
the  price  analysis  function.  How  rapidly  this  takes  place  will  depend  on 
the  effective  implementation  of  the  intelligent  manual  system  and  the 
requisite  knowledge  bases,  and  the  effectiveness  of  the  procedures  for 
maintaining  and  updating  the  system. 
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5.  FUTURE  RESEARCH  (SOW  4.3.3) 

In  this  chapter,  three  future  research  areas  related  to  the  proposed 
expert  system  are  identified.  They  are: 

1.  Hunan  interface  issues  and  related  implementation  problems. 

2.  System  extensions  to  include  cost  analysis  at  other  procurement 
levels . 

3.  Prototype  application  and  expansion. 

Each  is  discussed  in  further  detail,  and  the  issues  and  research 
required  to  deal  with  them  identified. 


3.1  Hunan  Interface  Issues 

The  original  statenent  of  work  paid  scant  attention  to  user  interface 
issues.  However,  it  has  bccone  apparent  from  our  investigation  that  these 
issues  were  one  of  the  najor  reasons  why  prior  Air  Force  systems  for 
supporting  price/cost  analysis  have  been  ineffective.  There  are  two  sets 
of  tasks  to  be  accomplished. 

1.  User  interface  problems  in  base  level  pricing  need  to  be 
evaluated  and  identified.  There  are  two  possible  orientations 
for  this  kind  of  study.  One,  the  problems  with  the  proposed 
intelligent  manual  can  be  studied.  Two,  the  problems  that 
currently  exist  with  other  AF  programs  (or  some  subset  of  these) 
can  be  studied. 

2.  The  intelligent  manual  needs  a  good  user  interface.  This  must 
be  designed  to  respond  to  the  human  interface  problems 
identified  earlier. 

3.1.1  Issues  in  Human-Computer  Interface  design 

A  number  of  different  classifications  of  human-computer  issues  have  been 
proposed  in  the  literature  [16,  15]  .  Many  of  these  criteria  take  the  form 
of  "rules-of- thumb"  —  they  define  characteristics  of  good  systems  and  are 
classified  in  some  natural  groups  [11,  7,  8].  For  example,  the  top  level 
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of  Che  list  in  Gebhardt  and  Stellmacher  is: 

1.  Simplicity. 

2.  Clarity. 

3 .  Uniqueness . 

4.  Comfortable  Language. 

5.  Other  comfort. 

6.  Evidence  and  Re-usability. 

7.  Stability. 

8.  Data  Security. 

For  each  of  these  items,  there  are  simple  design  criteria  that,  if 
followed,  should  result  in  a  good  interface.  For  our  purposes,  such  a  list 
of  rules  is  not  very  useful  as  it  does  not  identify  the  principles  behind 
the  items  in  the  list  and  therefore  the  application  of  the  rules  in 
different  systems  need  not  result  in  uniform  design. 

The  recent  book  by  Card,  Moran,  and  Newell  [1]  classifies  human-computer 
interface  design  issues  into  three  sets  of  structural  variables 
corresponding  to  the  three  structural  components  of  an  interface  (the  user, 
the  task,  and  the  computer  system),  and  four  sets  of  performance  variables 
characterizing  the  behavior  of  the  human-computer  system.  These  issues  are 
identified  in  Figures  12  and  13. 

We  may  also  view  the  problem  (for  price  analysis)  as  one  of  designing  an 
interface  to  an  information  system.  If  so,  a  list  like  the  one  defined  by 
Ramakrishna  [12]  would  be  appropriate.  This  list  is  an  attempt  to 
characterize  the  capabilities  that  the  system  must  possess  (and  can 
therefore  be  subsumed  within  the  System  Variables  component  of  the  list  in 

4 

Figure  12) . 

1.  Information  Structuring  Issues:  What  facilities  does  the  system 
provide  for  structuring  information? 
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Task  Variables 

User  Variables 

Computer  Variables 

Task  Domain 

Intellectual  Abilities 
General  Intelligence 
Technical  Ability 

Dialogue  Style 

TAsk  Model 

Cognitive  Style 

Risk  Preference 

Curiosity 

Persistence 

Command  Syntax 

Experience 

On  System 

Frequence  of  Use 

Naming  Conventions 

Knowledge 

Method  Knowledge 
Conceptual  Knowledge 

Task  Expertise 

Display  Layout 

Perceptual-Motor  Skill 
Typing  Rate 

Manual  Skill 

Input  Devices 

Response  Time 

Figure  12:  Human-Computer  Interface  design:  Structural  variables 


Basic  Measures 

Subjective  Measures 
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Functionality 

Acceptability 

Fatigue 
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Learning 

Time 
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Stress 

LTM  Recall 

Error 

Quality 

Robustness 

Figure  13:  Husian-Computer  Interface  design:  Performance  variables 


2.  Information  Distribution:  What  mechanisms  support  the  location, 
referencing,  and  distribution  of  information? 

3.  Level  of  Integration  of  the  User  Interface:  The  extent  to  which 
the  user  interface  is  flexible,  consistent,  and  uniform. 

4.  Development  and  Evolution:  Does  the  system  provide  a  strategy 
for  the  formal  and  informal  development  of  the  system  under  user 
guidance? 

5.  External  representation  Issues:  Does  the  system  provide  a 
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mechanism  for  automated  and  other  input/output  of  information? 

These  issues  are  slightly  different  from  those  described  earlier;  however, 
they  may  be  relevant  to  the  discussion  in  the  context  of  price  analysis. 

The  term  "user-friendly"  captures  the  essence  of  the  concept  that 
systems  should  not  intimidate  users  with  their  complexity  or  with  their 
idiosyncratic  organization  of  a  task  that  the  user  believes  can  be 
performed  in  other  ways.  If  there  is  a  comfortable  alternative  to  the 
computer  system,  the  user  will  prefer  that  alternative  rather  than  face  an 
unfriendly  system.  The  above  layout  of  interface  design  issues  essentially 
categorizes  all  that  is  now  known  about  user-friendly  design. 

5.2  Extension  to  Cost  Analysis 

The  intelligent  manual  approach  could  be  expanded  to  include  cost 
analysis.  The  cost  analysis  task  is  more  structured  and  thus  might  be  more 
amenable  to  a  deductive-style  of  expert  system  design.  This  extension  will 
test  the  system's  capabilities  in  the  Task  Action  component  and  extend 
beyond  the  current  spreadsheet  capabilities  of  COPPERIMPACT . 

Including  cost  analysis  within  the  capabilities  of  the  system  projects 
its  use  by  central  procurement  activities .  This  procurement  environment  is 
significantly  different  from  the  base-level  procurement  environment.  The 
following  major  tasks  would  need  to  be  accomplished: 

1.  Collecting  materials  on  how  various  subtasks  of  central 
procurement  are  to  be  performed. 

2.  Identifying  the  different  kinds  of  cost  analysis  and  how  they 
might  be  done  by  an  expert  system. 

3.  Identifying  different  tasks /sub-tasks  in  the  accounting/auditing 
domains  and  determining  the  ways  to  accomplish  them. 

4.  Determining  what  kinds  of  expert  systems  (or  human  interfaces) 
are  appropriate  for  the  tasks. 

A  prototype  expert  system  would  have  to  be  designed  capable  of 
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performing  some  seLected  set  of  cost  analysis  tasks.  This  system  could  be 
integrated  into  the  current  prototype  intelligent  manual.  In  implementing 
the  selected  subtasks,  studies  of  price  analysts  performing  those  subtasks 
(using  protocol  analysis)  would  be  undertaken.  Thus  the  implementation 
task  will  require  two  stages:  data  gathering  and  system 
design/ implementation.  Cost  analysis  cases  are  generally  better  "cases" 
than  price  analysis  cases;  thus,  there  is  a  higher  likelihood  that  protocol 
analysis  could  be  successfully  carried  out.  Two  subtasks  of  cost  analysis 
suggest  themselves  as  ideal  for  implementation: 

1.  Developing  a  profit  objective,  and 

2.  Determining  the  applicability  of  Cost  Accounting  Standard  (CAS) 

414  (Facilities  and  Cost  of  Money). 

5 .3  Extension  and  completion  of  current  prototype 

The  current  prototype  intelligent  manual  only  covers  one  aspect  or  type 
(Competitive  Pricing)  of  base~level  price  analysis.  There  are  two  other 
aspects  of  price  analysis  that  the  system  discusses,  but  does  not  currently 
cover.  They  are  Published  Prices  and  Secondary  Comparisons.  Also,  though 
the  intelligent  manual  attempts  to  establish  the  procurement  context  within 
vhich  the  buyer  functions  and  provides  some  support  for  procurement 
activities,  the  system's  does  not  fully  support  these  capabilities. 

Attempting  to  establish  system  use  in  a  realistic  setting  would  raise 
human  interface  issues  identified  earlier  as  well  as  issues  concerning  the 
efficacy  of  the  knowledge  base  and  databases  associated  with  the  system. 
Other  issues  such  as  the  availability  of  hardware,  the  nature  of  the 
hardware,  the  needs  for  networking  between  buyers  and  price  analysts,  the 
need  for  online  information  as  well  as  help  would  all  have  to  be  addressed. 
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6.  LIMITATIONS  OF  PROPOSED  SYSTEM  (SOW  4.3.3) 

This  chapter  discusses  the  limitations  of  the  current  prototype 
intelligent  manual. 

Price  Analysis  is  only  a  small  component  of  the  overall  procurement 
activity  of  the  buyer.  Other  problem-solving,  information  gathering,  and 
decision  making  activities  must  be  carried  out.  Most  of  these  other 
activities  are  less  amable  to  "expert  systems"  technology  applications. 
However,  the  user  can  be  supported  to  varying  degrees  while  performing 
them.  For  example,  a  system  for  training  negotiators,  or  a  system  that 
interviews  the  user  of  an  item  proposed  for  purchase  to  determine  the 
technical  specifications  (or  aids  the  buyer  in  the  interview)  may  be  as 
useful  as  a  price  analysis  system.  The  ZOG-based  intelligent  manual 
approach  would  be  a  first  step  in  the  design  of  such  systems. 

The  primary  limitations  of  the  proposed  prototype  system  relate  to  the 
restricted  focus  of  the  study.  The  task  support  and  guidance  systems  that 
interact  most  heavily  with  the  buyer  contain  no  reasoning  capability.  They 
provide  the  user  with  structure  and  assistance  for  making  decisions,  but  do 
not  draw  inferences  based  on  prior  decisions.  The  human-machine  interface 
is  not  as  well  developed  as  desirable.  As  discussed  earlier,  this  may  be  a 
major  impediment  to  successful  implementation  of  the  system. 

The  system  is  designed  for  base-level  procurement.  Modifications  needed 
to  apply  the  system  in  other  procurement  environments  must  be  identified 
and  incorporated  into  the  system.  Including  cost  analysis  capabilities  as 
part  of  the  system  is  an  example  of  such  an  extension.  The  prototype  also 
is  directed  primarily  at  buys  of  between  $10,000  and  $500,000.  These 
differ  significantly  from  the  excluded  buys. 

The  system  has  not  been  tested  with  respect  to  actual  expert  behavior. 
Two  issues  are  involved  here.  First,  this  kind  of  testing  requires  that  the 
prototype  be  implemented  and  subjected  to  the  tests  of  the  work  place.  The 
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7.  ALTERNATIVE  SOLUTIONS  (SOU  4.3.4) 

la  this  chapter  we  identify  several  alternative  solutions  that  can  also 
enhance  the  procurement  process.  These  include  alternative  expert  systems 
for  computer-aided  cost /price  analysis.* 

7.1  Alternative  Structures  for  the  Problem 

Two  expert  system  alternatives  are  to  construct  a  simple  deductive 
system  or  to  construct  a  data-rich  knowledge  manipulation  system.  Both 
alternatives  possess  a  higher  level  of  "intelligence"  than  does  the 
proposed  expert  manual. 

7.1.1  Simple  Deductive  Systems 

A  system  capable  of  making  simple  deductions  on  its  own  requires  some 
reasoning  capabilities.  One  way  of  designing  such  a  system  is  to  construct 
a  subsystem  containing  a  control  structure,  one  having  a  capability  for 
making  historical  deductions  given  specified  relationships,  and  specialists 
that  can  be  called  when  specific  expertise  is  needed.  In  the  case  of  price 
analysis,  the  specialists  would  include  accounting,  management,  auditing, 
and  overhead  allocations.  The  user  would  be  required  to  provide  the  system 
with  input  data  and  also  respond  to  requirements  which  could  not  be  handled 
by  the  system.  These  circumstances  would  most  likely  be  encountered  where 
the  buyer  either  interfaces  with  other  specialists,  needs  information  not 
contained  in  the  system's  data  base,  or  where  unstructured  reasoning  is 
required  to  solve  the  problem.  Note  that  this  alternative  assumes  that 
these  other  specialists  will  be  available  online  —  either  live  or  in  the 
form  of  an  expert  computer  system. 

The  major  impediment  of  this  alternative  is  that  the  data  bases  and 
other  support  requirements  are  not  available.  However,  current  technology 
appears  to  be  adequate  to  construct  such  a  system  within  the  procurement 
environment  given  the  developmental  resources. 
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7 .1 .2  Data-Rich  Knowledge  Manipulation  Systems 

A  data  rich  system  would  be  the  ultimate  in  intelligent  systems  for 
contract  price  analysis.  It  would  have  the  capability  to  read  the 
contractor's  proposal  and  based  on  the  historical  and  analytic  data  in  the 
system,  analyze  the  contract  and  formulate  a  government  position.  This 
requires  that  the  system  have  access  to  the  technical  specifications  of  the 
buyer,  a  history  of  the  activity  in  purchasing  related  items,  a  history  of 
the  contractor's  performance  on  other  contracts,  a  history  of  similar 
contractors  and  similar  buys  as  well  as  the  logical  reasoning  capabilities 
required  to  analyze  the  the  current  contract  in  light  of  these 
considerations . 

Implementing  such  a  system  would  require  very  good  analytical  tools  as 
well  as  data-base  capabilities  currently  not  available  (and  not  expected  to 
be  available  for  a  considerable  period  of  time).  This  is  the  most  viable 
alternative  in  the  very  long-term  (25  years  or  more).  Realistically,  such 
a  system  is  unlikely  to  develop  in  a  systematic  way  but  would  probably 
evolve  in  some  manner  from  existing  piece-meal  systems . 

7.1.3  COPPER  IMPACT  Baaed  System 

An  expert  system  using  COPPER  IMPACT  as  its  basis  is  not  a  viable 
alternative.  COPPER  IMPACT  does  not  possess  the  requisite  capabilities  for 
such  a  task.  It  is  primarily  a  set  of  programs  with  statistical  and 
spread-sheet  capabilities.  As  explained  earlier,  it  would  be  useful  as  a 
component  of  the  Task  Action  system  in  the  intelligent  manual,  or  as  an 
auxiliary  system  to  another  expert  system.  Likewise,  superimposing  an 
expert  system  tutor  on  top  of  COPPER  IMPACT  would  be  of  little  benefit 
given  the  limitations  of  COPPER  IMPACT  system  and  its  documentation.  One 
of  the  major  faults  with  COPPER  IMPACT  has  been  its  user-unfriendliness 
—  superimposing  an  expert  system  on  top  of  this  would  not  change  this 
primary  problem. 

Other  systems  currently  used  by  procurement  personnel  (CIAPS,  etc.) 
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provide  data  bases  that  would  be  useful  for  any  expert  system.  However, 
none  of  these  were  designed  to  provide  a  basis  for  constructing  expert 
systems.  They  occasionally  share  COPPEB  IMPACT'S  unfriendliness;  in 
general,  they  all  tend  towards  inflexibility  in  command  languages. 

7 .2  Non-expert  Systems 

Non-expert  systems  which  provide  online  assistance  would  be  helpful  to 
procurement  personnel.  Attempts  are  currently  being  undertaken  in  several 
areas  such  as  online  contract  writing,  procurement  history  maintenance, 
electronic  mail,  and  others.  The  human  factors  implications  of  these 
implementations  need  to  be  evaluated  and  improved.  If  designed  carefully, 
these  systems  might  provide  the  basis  for  building  future  expert  systems 
within  the  procurement  environment . 
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III.  LIST  OP  THE  CASE  TYPES  REQUESTED  AHD  THE  CASE  TYPES  RECEIVED. 

Butd  on  our  investigations,  we  claaaified  procurement  situation!  into  a 
small  set  of  categories  and  requested  a  small  number  of  cases  for  a 
selected  set  of  categories.  The  classification  is  presented  below  with  the 
number  of  cases  requested  in  each  category  and  the  number  received  in  that 
category  in  parentheses.  Our  study  was  focused  on  base** level,  intermediate 
size  purchases,  and  so  we  did  not  desire  any  cases  from  the  very  low  end  or 
the  high  end  of  the  size  spectrum.  We  determined  the  numbers  of  cases  we 
wanted  to  receive  by  roughly  estimating  their  relative  frequency  (based  on 
discussions)  and  setting  a  maximum  limit  on  the  number  of  cases  ve  expected 
to  show  buyers  and  price  analysts.  We  presume  that  the  numbers  we  actually 
received  represent  the  actual  frequency  of  these  cases. 

It  will  be  apparent  that  we  received  many  cases  in  categories  that  were 
not  appropriate  for  this  study,  and  did  not  receive  sufficient  cases  in 
certain  categories.  Initially,  we  attempted  to  ensure  that  we  had  a  well** 
distributed  set  of  complete  cases;  however,  the  wait  for  complete  cases  in 
appropriate  categories  to  be  accumulated  was  seriously  jeopardizing  the 
time  schedule  for  the  project  and  we  decided  to  go  ahead  with  what  we  had. 
(The  first  number  in  the  parenthesis  indicates  the  number  of  cases 
requested  the  second  indicates  the  number  received.) 


A.  0  to  $1000:  Ho  price  justification  required  (0,  0). 

B.  $1000  to  $25,000:  Price  Analysis  only;  Buyer  cannot  ask 
for  cost  data  (4,  12). 

termadiata  Purchases  ($25,000  to  $500,000) 


A.  Price  Analysis 


1.  Competitive 


a.  Actual  Competition  (2,  2) 
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b.  "Based  on"  Competition  (2,  0) 

c.  Competitive  Environment  (2,  0) 

2.  Catalog  Prices 

a.  Actual  Catalog  Price  <2,  2) 

b.  Based  on  Actual  Catalog  Price  (2,  0) 

3.  Comparison  to  Prior  Purchases  (4,  4) 

4.  Government  Estimate 


a.  Detailed  Estimate  (2,  1} 

b.  Lump-Sum  Estimate  Cl,  0) 

5 .  GSA  Schedules 

a.  Mandator;  (1,  1) 

b.  Non-Mandator;  (2,  2) 

6.  Value  Anal;sis  (1,  0) 

B.  Cost  Anal;sis 

1 .  General  Cases :  Buyer  must  decide  if  DD  Form  633  is 
to  be  used  (1,  4). 

2.  Special  Cases 
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B.  Cost  Analysis 

1.  General  Case  (0,  0) 

2.  Special  Cases 

1.  Construction  Hods  (0,  4) 

2.  8A  Set  Asides  (0,  0) 

3.  Service  Contract-Act  Price  Adjustments  (0,  0) 

4.  Packaging  Costs  (0,  0) 

During  an  interview  with  an  analyst,  the  interviewers  used  the  following 
brief  description  of  a  case  packet  as  a  reference.  We  provide  this  as  a 
description  of  a  typical  case  in  lieu  of  a  sample  case  packet. 


CASE  HO.  6  BASE  LEVEL  PRICING 
Type:  GSA  non-mandatory  schedule 


Contract  Ho.:  F33601-83-F3563 
Item:  Signal  generator 


Contents : 


1.  Order  for  supplies  or  services  (SD1155) 

2.  Request  for  proposel 

3.  Requisition  (DD1348-6) 

4.  Specifications— sppesr  to  he  pages  out  of  the  offeror's 
catalog 

3.  Computer  printout  of  something  that  looks  like  an  abstract 
of  bids  —  appears  that  another  contractor  was  contacted 
but  was  not  able  to  give  a  quote  because  they  do  not  carry 
this  model.  Seems  to  serve  as  a  PNM. 
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Hamarka : 


1.  Ho  PHK 

* 

2.  Only  one  bid  but  buyer  atteapted  to  secure  another  bid  from 
one  other  supplier.  The  other  supplier  could  not  meet 
specifications  end  therefore  could  not  submit  a  bid. 

3.  Appears  that  the  contract  number  changed  in  process.  No 
mention  of  why  this  might  have  happened.  Possibly,  the 
first  attempt  to  secure  a  proposal  was  not  satisfactory 
ifor  instance,  all  the  bids  may  have  been  too  big,  or  no 
bids  were  received)  and  the  solicitation  had  to  be  redone. 

4.  pie  price  from  the  offeror's  catalog  appears  to  be 
$7900.00,  however,  "he  contract  price  is  $7680.  What  is 
the  reason  for  this  difference?  This  is  not  mentioned  in 
the  material  provided.  Most  probably,  the  company  gives 
the  government  a  42  discount  if  the  buyer  asks  for  it. 
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IV.  WRITTEN  INSTRUCTIONS  FOB  INTERVIEWS 

b 

h 

Ths  objective  of  this  study  is  to  understand  your  methods  for  analyzing 
proposals  to  determine  a  fair  and  reasonable  price  and  for  selecting  a 
proposal. 

We  will  do  this  by  shoving  you  a  set  of  cases.  In  each  case,  we  will 
give  you:  the  Purchase  Bequests  and  other  information  relevant  to  the  case; 
the  bids /proposals  from  contractors  and  documentation  relevant  to  these 
bids.  As  you  go  through  these  cases,  please  talk  aloud  about  the  steps  you 
are  performing,  the  assumptions  you  are  making,  or  the  procedures  you  are 
following.  If  you  need  any  additional  information,  you  can  ask  us,. and  if 
we  have  it,  we  will  give  it  to  you. 

Some  of  the  case  packets  that  we  will  give  you  are  incomplete;  this  is 
NOT  a  test  of  your  ability  to  detect  incomplete  cases.  The  cases  are 
incomplete  because  we  were  not  able  to  get  complete  information  on  these 
cases  and  we  would  like  to  identify  the  missing  information  and  the 
procedures  you  follow  to  obtain  such  information  or  the  procedures  you 
follow  in  the  total  absence  of  such  information. 

Tour  analysis  will  be  conducted  in  close  interaction  with  us  —  please 
talk  as  much  as  possible.  Ideally,  we  would  not  interfere  in  your 
thinking;  however,  we  will  encourage  you  to  talk,  if  necessary  by  asking 
questions. 

You  should  continue  your  analysis  to  the  point  where  you  can  complete  a 
PNM.  We  include  a  copy  of  AFLC  Form  1612  (WFAFB  Form  246)  with  each  case 
to  use,  in  case  vou  wish  to  do  so.  You  are  welcome  to  use  plain  paper,  if 
you  vish  (you  may  prefer  an  alternate  form  but  it  is  unlikely  that  we  will 
be  able  to  procure  it  for  you). 
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